After completing this lesson, you’ll be able to:
FME Flow has a model-driven architecture because its processes are expressed as models. In FME, these models are better known as workspaces.
Workspaces are created – we call it “authored” – using FME Form. In particular, the FME Workbench application is used. FME Workbench is a client of FME Flow, and so they form a client-server pair. However, both share the same core engine and process data in the same way.
Servers are applications that provide a resource or service.
Clients make requests of servers to access a resource or service.
Together, a client-server pair form a distributed application structure. Usually, the client and server exist on separate hardware and communicate over computer networks, but sometimes both the server and client exist on the same system.
Let's make sure you get the terminology right. FME Form is the desktop program that includes the FME Workbench, FME Data Inspector, and FME Quick Translator applications. Workspace authoring occurs in the application called FME “Workbench,” but the process defined in the canvas window of FME Workbench is called a “workspace.” The terms are so similar that they are easily confused.
Because FME Workbench is a client of FME Flow, it may be used to transfer authored workspaces to and from FME Flow. We call this transfer publishing.
FME Workbench has the ability to:
The ability to transfer a workspace back to FME Workbench means workspaces can be downloaded for editing and maintenance and then published back to FME Flow.
Workspaces are stored on FME Flow in devices called repositories. Each FME Flow may have multiple repositories, but any workspace can only belong to one of them, without creating a duplicated workspace.
A repository consists of two parts:
1) The .fmw files from any workspaces that have been published to FME Flow are stored in the file-based part of the repository.
2) Metadata related to the workspace is held separately in the FME Flow Database. This metadata includes information about the contents of the workspace; for example, source and destination datasets, workspace feature types, and published parameters.
Repositories are managed by the FME Core. They can be accessed (by authors and administrators) through the FME Flow Web Interface.
Security in FME Flow is very important, and never more so than for repositories.
You can think of each repository as being like a folder on a file system, with the same ability to grant access rights to individuals and groups. So, for each repository you create, be sure to check the security permissions. If you don’t, then end-users may not get access to the repository!
The functionality for publishing or downloading workspaces is accessed in FME Workbench either through the menubar:
... the toolbar:
... or the Start Tab:
Workspaces can also be uploaded and downloaded directly through FME Flow on the Workspaces page:
The Publish to FME Flow tool in FME Workbench opens a simple wizard interface, the first dialog of which defines a connection to FME Flow.
Selecting Connect to FME Flow opens a dialog with fields in which to define connection credentials. These connection details are saved so that they can be reused in the future simply by picking from the drop-down list:
The next dialog defines the repository in which to store the workspace:
Either an existing repository can be used, or a new one created. The workspace name can also be edited, even making it different to what it is saved as locally.
This dialog only appears when there are databases and/or web connections that need to be uploaded with the workspace.
This workspace contains both a database connection and a web connection that need to be uploaded to function on FME Flow.
Web connections require authentication in FME Flow in addition to adding the web service to FME Flow.
The database connection requires no further authentication, but care must be taken not to accidentally overwrite an existing database connection with the same name that might already be defined on FME Flow.
The final dialog defines which service(s) the workspace is to be registered against. A workspace may be registered for use with any number of these services.
The Job Submitter service allows FME Flow to run a workspace as-is. This is the closest to running a workspace in FME Workbench. All inputs and outputs are defined in the workspace, so data is simply written out and not streamed or delivered in any other manner.
Job submission is ideal for testing workspaces, writing to databases, and for running large-scale and batch translations that make use of the server process queue.
You can learn about the other services in the Documentation or the Methods for Running a Workflow course.
Once a workspace has been published, the republish tool becomes active. Further updates to the workspace (within the same FME Workbench session) can then be uploaded with a single click.
The same parameters are used as before. If changes need to be made to these parameters, then the full publishing wizard should be used.
Workbench can also download a workspace held in an FME Flow repository. This is usually done in order to make edits to the workspace. Note that downloaded workspaces are copies of the original, which remains in the FME Flow repository.
The two parts of the downloading wizard are the FME Flow connection, which is the same as the publishing wizard, and the repository and workspace selection tool:
The user is then prompted for a location to save the workspace. The default (on Windows) is <User>\Documents\FME\Workspaces. The workspace and any resources are then downloaded and saved to that location.
Once downloaded, the workspace is automatically opened within FME Workbench for editing.
Besides workspaces, it’s also possible to publish/download FME custom transformers and custom formats to and from an FME Flow repository.
When the source data for a translation is stored as files (rather than a feed or database), it is possible to publish data to an FME Flow repository along with the workspace.
This data upload method is fast and simple but limits future access to the data. A better practice is to publish your data to the Resources folder separate from the workspace. Learn how to use the Resources folder in the Connect Data to FME Flow course.
In this workspace the source dataset is MapInfo TAB:
A MapInfo TAB dataset is made up of a series of files (.tab, .dat, .id, .map). When this workspace is published the wizard allows us to publish the data files alongside it by simply checking the box labeled Upload data files.
FME automatically selects the files to upload based on what it thinks is necessary to run the translation. If there are other files you wish to upload, or files FME selected that you don't wish to upload, the Select Files button allows you to make changes:
This dialog also allows you to change where the files are published, but for now, we'll ignore that setting and go with the default of publishing to the repository.
Once the publishing wizard is complete, those files are uploaded to FME Flow and tagged for use with this workspace.
